Electrophysiological Feedback in Adaptive Human Computer Interfaces

This paper explores the feasibility of using EEG in the context of Stimulus Rich Reactive Interfaces (SRRI’s), as a source of feedback on the cognitive state of the user. We have run experiments to evaluate the utility of two potential EEG measures of whether a stimulus has been perceived: 1) reduced EEG power in the alpha band at posterior brain areas and 2) a P3-like positive deflection over parietal areas. Such measures would enable re-presentation of a critical stimulus that has been missed. This paper considers whether, in the context of SRRIs, these measures can be reliably extracted online, i.e. in real-time. To determine this, we have investigated the extent to which online extraction of these measures predicts target report. Our results are positive, and suggest that a combination of multiple approaches can be used to improve the reliability of single trial discrimination. We also discuss possible ways in which such a system could be implemented and integrated into a head-mounted wearable display that is electrically isolated from the surroundings by optical input and output. Introduction One of the fundamental problems involved in real-time processing is information overload. A system has to pick and choose which stimuli to attend, before it has expended the resources to process them. The human brain has evolved a sophisticated attentional mechanism to pick and choose what information is selected for further processing. As a consequence, the designer of a computer interface is not always able to ensure that what the interface knows is important (such as a low altitude warning on an aircraft) is selected by the human user’s attention system. HCI design has learned a great deal of tricks to increase the inherent salience of input signals to a human user (using warning lights and sounds, for example), but accidents still occur due to critical information that is ignored. If described using computer networking as a metaphor, this situation is akin to transmitting information over a communication channel without acknowledgement of receipt. The computer is transmitting information to the user with the intent that it be received, but the system has no way of verifying that this transmission was successfully completed. No well-designed computer-computer interface operates in this way. Acknowledgement of packet receipt is a core element of any modern networking technology. The goal of this research is to record an acknowledgement signal from scalp recorded brain waves. Electrophysiological neuroscience has identified a number of electrical signals produced by the brain when information is encoded into working memory that could serve this role. In this paper, we explore a novel use of recorded brain signals, to enhance the reliability of information transmission from computer to user. Brain Computer Interface: The direct brain to computer interface (e.g. Vidal 1973, Levine, et al 2000, Beverina, Palmas, Silvoni, Piccione, & Giove 2000) has been one of the most useful applications to have arisen from electrophysiological neuroscience. By recording brain waves from human subjects with motor impairments, it has been possible to give the severely disabled a means of communication with the world (e.g. Cilliers & Van Der Kouwe 1993) Using these techniques, a subject can passively view a matrix of characters and guide a computer interface that is recording and analyzing EEG signals to the intended words he or she wishes to communicate. For some people, the electrical signal produced by their brain is 1 The full reference for this article is, B. Wyble, P. Craston and H. Bowman. "Electrophysiological feedback in adaptive human computer interfaces." Technical Report 8-06, Computing Lab, University of Kent at Canterbury, September 2006. the only means by which they can consciously communicate with the world. The Brain-ComputerInterface (BCI) field is now quite mature, using cutting edge technology to maximize the accuracy and speed of the information that can be read from the disabled subject’s brain (Meinicke, Kasper, Hoppe, Heumann & Ritter 2003; Hochberg, et al 2006) Work in the BCI field has almost exclusively focused on human-to-computer information transfer. However, recording brain signals can also serve to enhance the reliability of information flow in the opposite direction, an application that is virtually unexplored. We will describe a system that can use brainwaves to warn a computer that its user may have missed a critical piece of information. This warning will allow the computer to re-present missed information until it is perceived. The algorithms described in this report are simpler than those commonly used in BCI research. The intent of our approach is to develop a device with has the following characteristics: o Implementable with currently available electrical components o Small enough to be enclosed within a helmet o Easily shielded from nearby interference o Power consumption sufficiently low to run on lightweight battery power o Extremely rapid response (response within less than 1 second of target onset) o Easy to set up These restrictions presently rule out both sophisticated waveform analyses and multi-electrode arrays. This paper focuses on whether an uncomplicated analysis of data recorded from a single pair of electrodes can provide useful information to the computer about whether a target was detected. We will focus on two elements of the human EEG signal as indicators of target perception: the P3 component of the Event Related Potential (ERP) and changes in the power of oscillations near 10 hz, known as the Alpha power band. The P3 Component: When one records EEG from the human scalp, the signal measured is deflected by ongoing cognitive operations. In the EEG literature, these deflections are referred to as components, which are observed in the Event Related Potential (ERP) that emerges from averaging together a large number of trials, time locked to the onset of a salient stimulus. ERP components can be manipulated experimentally; hence, they have been associated with particular cognitive processes. Whereas the early part of the waveform is associated with sensory processing of target stimuli, the later part is associated with high-level processing of a stimulus. This component is typically called the P3 (i.e., the third positive peak of the ERP, also referred to as the P300 due to its typical latency of 300ms post-stimulus). Although some researchers have identified a frontally located P3a component, which is elicited by infrequent but task-irrelevant stimuli, we focus on the P3b, which has its maximum over parietal electrode sites. The P3b (called P3 from here on) is present for stimuli that are both infrequent and relevant to the task (Squires et al. 1975), As the P3 is only observed in the ERP for target stimuli that are detected by the subject (Vogel et al. 1998), it can be assumed to be an indication of an item being encoded into working memory (Donchin, 1981). Depending on the amount of noise in the signal, one normally has to average across a considerable number of trials to obtain a clean ERP waveform. However, the P3 component is often large enough to be detected even in the raw EEG. Of course, one cannot draw conclusions about P3 latency and shape from raw P3s; however, they are often clear enough to be detected on a trial-by-trial basis. The algorithm used in our approach focuses on these raw P3s. Alpha Waves: During scalp EEG recording, alpha oscillations (in the 8-12 hz band) are prominent in posterior areas of the brain and this is especially true when the subject’s eyes are closed. Alpha activity is so large that it can easily be discerned in the raw, unfiltered signal and is generally much larger than the P3 component, within an individual trial. However, because alpha activity is poorly phase locked to external events, it is not visible in most event triggered average plots, as employed in ERPs. However, if alpha activity is computed within individual waveforms, using FFT’s, strong changes in this oscillation can be observed in relation to cognitive events. One prominent relationship between oscillations and mental activity is that alpha power tends to diminish at the onset of a cognitive event, such as detecting a target, encoding something into memory, or initiating a movement. This is called an ERD or Event Related Desynchronization (see Pfurtshellter & da Silva 1999 for a review). It is thought that alpha activity may reflect an area of cortex that is being held in idling state. Experiments and Data: To test our ideas, we used data from another experiment, which presented targets embedded in a stream of distractors. We were able to record prominent P3 ERP’s as well as alpha waves. This EEG data serves as a test of the ability of online algorithms to determine wether a target was seen or missed. The experiment involved Rapid Serial Visual Presentation (RSVP), a presentation method commonly used in the visual attention field (Weichselgartner & Sperling 1987). In RSVP, a subject fixates on the center of the screen and views a stream of constantly changing stimuli, which contains one or more targets. In the experiments reported here, subjects viewed a stream of digits in search for any uppercase letter, which was reported at the end of a trial. The subjects did not know when the letter would appear, and stimuli changed every 47 ms. EEG data were compared with target detection on corresponding trials, to gauge the ability of EEG to discriminate between target seen and target missed trials for a random group of 12 subjects. Method Participants: Twelve underand postgraduate university students (mean age 24.1, Std. Dev. 2.9; 6 male and 6 female; 11 rightand 1 left-handed) provided written consent and received 10 GBP for participation. Participants were free from neurological disorders and had normal or corrected-to-normal vision. The study was approved by the local ethics committee. Stimulus presentation: We presented alphanumeric characters (Arial font, 5cm mean height) at a distance of 100cm (2.86 visual angle) on a 21" CRT computer screen (1024x768 @ 85Hz) using the Psychophysics toolbox (Brainard 1997) running on Matlab version 6.5 under Microsoft Windows XP. A photodiode verified exact stimulus presentation timing. Participants viewed four blocks (3 RSVP/1 skeletal) of 100 trials. Within each block, there were 96 trials containing a single target and four catch-trials consisting only of distractors. Five practice trials preceded the first block in both the RSVP and skeletal conditions. The underlying structure and timing of RSVP and skeletal streams was equal. However, whereas in RSVP, the target was embedded into a continuous stream of distractors, unmasked skeletal streams contained only the target and masked skeletal streams consisted of the target and a following distractor. The target for each trial was chosen at random from a list of 14 capital letters (’B’,’C’,’D’,’E’,’F’,’G’,’J’,’K’,’L’,’P’,’R’,’T’,’U’,’V’); distractors could be any digit except ’1’ or ’0’. The target item’s position in the stream varied between 10 and 55. A fixation cross presented for 500ms preceded each stream. Stimuli were presented at a rapid rate of approx. 20 items per second (SOA 47.1ms; no inter-stimulus interval) to ensure participants missed a considerable number of targets. The RSVP stream was made up of 70 items (total stream length 3.3 seconds); Each stream ended with a dot or a comma (present for 47.1ms) as final item. Following stream presentation, subjects used a computer keyboard to enter the target letter or press space if they had not seen a target. Subsequently, they pressed dot or comma depending on what the last item of the stream was. The second task was included to ensure that subjects maintained their attention on the stream after the target had passed. Recording & Analysis: Electroencephalographic (EEG) recording was made using a Quickamp (BrainProducts, Munich, Germany) amplifier, with a 22-bit analog-to-digital converter. The sampling rate was 2000Hz and the data was digitally filtered at low-pass 85Hz and high-pass 0.5Hz at recording. Twenty electrodes were placed at standard locations according to the international 10/20 system Jasper(1958). Electrooculographic (EOG) activity was recorded from below and to the right side of the right eye. The data was referenced to a common average online and re-referenced to linked earlobes offline. Left mastoid acted as ground. Eye movement artefacts were removed by rejecting data in the window of 200ms before and after an eye blink. Subsequently, all data was inspected for sudden high and consistently low activity and data from 500ms prior to 500ms after an artefact was marked as bad and removed from further analysis. In total, 17% of RSVP and 20% of skeletal trials had to be excluded due to artefacts. For this paper, we focus on the data from two regions of interest, namely occipital (channels O1, O2) and parietal (channels P3, Pz, P4) regions for the study of early components and the P3 wave of the ERP respectively. After segmentation and averaging, an 8Hz low pass filter was applied to enhance visualisation of ERP components. EEG data was analysed using the BrainProducts Analyzer software and the Matlab EEGLab toolbox (Delorme & Makeig 2004). Signal Detection: For each trial, an algorithm determined whether subjects did or did not see a target based on the EEG data after the time of the target presentation. P3 amplitude: A measure of total area under the curve was computed for each participant (Figures 1,2), centered around the time of maximal P3 amplitude. This time window ranged, at most, from 300-700 ms after the target, but varied for each individual subject. This measure was taken for both target seen and target missed trials. A threshold value for each participant was set at 50% of the area under the curve from the average of all target-seen trials. Then, for each trial, we determined if the P3 exceeded this value. If a target-seen trial had a P3 of larger area than the threshold, the value was counted as a hit, otherwise the trial was scored as a miss. On target-missed trials, if the P3 area was larger than this value, the trials was scored as a false alarm, otherwise it was a correct rejection. With these measurements of percent hits and percent false-alarms, we were able to compute a d’ score, individually for each subject McNichol, D (1972).